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O ■ Abstract 



OO , Distributed Compressive Sensing (DCS) IT] improves the signal recovery performance of multi 

(N 

signal ensembles by exploiting both intra- and inter-signal correlation and sparsity structure. However, 



the existing DCS was proposed for a very limited ensemble of signals that has single common infor- 
mation lH]. In this paper, we propose a generalized DCS (GDCS) which can improve sparse signal 
detection performance given arbitrary types of common information which are classified into not just 
full common information but also a variety of partial common information. The theoretical bound on the 
required number of measurements using the GDCS is obtained. Unfortunately, the GDCS may require 
much a priori-knowledge on various inter common information of ensemble of signals to enhance the 



> 

(N 
(N 

in_ 

\ performance over the existing DCS. To deal with this problem, we propose a novel algorithm that 



can search for the correlation structure among the signals, with which the proposed GDCS improves 
(N. detection performance even without a priori-knowledge on correlation structure for the case of arbitrarily 

correlated multi signal ensembles. 

X. 

, Index Terms 

^ [ 

Compressive sensing, distributed source coding, sparsity, random projection, sensor networks. 

I. Introduction 

Generally, signals in various applications can be represented as sparse coefficients with a 
particular basis, meaning a signal vector x G has only K <^ N nonzero coefficients. Many 
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compression algorithms exploit this sparse structure, including MPS JPEG [|3l and JPEG2000 
BH. Compressive sensing (CS) is an emerging signal acquisition technique that has an advantage 
of reducing the required number of measurements for recovery of sparse signal. If a target signal 
X G is represented as a sparse signal with a particular sparse basis, one can recover it with 
only M < N measurements. It is known that the signal can be recovered with overwhelming 
probability if the sparsity K (simply, the number of nonzero elements) of the signal satisfies 
K < C{\ogNy^M [|5l, where C is a constant. 

Baron et al. [[D introduced Distributed Compressive Sensing (DCS), which exploits not just 
intra-, but also inter- joint sparsity to improve the detection performance. They assume the 
scenario of a Wireless Sensor Network (WSN) consisting of an arbitrary number of sensors and 
one sink node. In this scenario, each sensor should normally carry out the compression in a 
distributed way without cooperation of the other sensors and transmit the compressed signal to 
the sink node. At the sink node the received signals from all the sensors are reconstructed jointly. 
Here, a key of the DCS is the concept of joint sparsity, defined as the sparsity of the entire signal 
ensemble. Three models have been considered as joint sparse signal models in In the first 
model, not only each signal is individually sparse, but there are also common components shared 
by every signal, called common information, which allow reduction of required measurements by 
joint recovery. In the second model, all signals share the supports, the locations of the nonzero 
coefficients. In the third model, no signal is sparse itself, nevertheless, they share the large 
amount of common information, which makes it possible to compress and recover the signals. 
While the second model, called the Multiple Measurement Vector (MMV) setting, has been 
actively explored in BH-jHl, to the best of authors' knowledge, the first model has been studied 
for only a limited ensemble of signals that has single common information. 

Despite its limitation, the first model is applied in many other applications such as flU, [[TOl as 
well as in the WSN (HI. In |l9l, they extract a common component and an innovation component 
from various face images for facilitating an analysis task such as face recognition. In [10], 
when implementing image fusion that combines multiple images of the same scene into a single 
image which is suitable for human perception and practical applications, they model the constant 
background image as common information and the variable foreground image as innovation 
information for efficiency of the process. 

However, it is unrealistic to assume that there exists only common information. Practically, in 
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most of situations, partial common information, which is firstly proposed in our conference 
version paper [[TTIl . as well as full common information are measured by arbitrary number 
of multiple sensors. Using this notion, we introduce partial common information leading to 
a generalized DCS (GDCS) model in this paper and obtain the theoretical bound of the number 
of measurements for exact reconstruction. However, to take advantage of partial common infor- 
mation, the decoder should know partial common structures of signals, which is not typically 
known to the decoder. To deal with this problem, we also propose a novel algorithm that can find 
the correlation structure among sensors to help decoder exploit partial common information. This 
algorithm can provide significant performance improvement. In summary, the main contributions 
of this paper are as follows. 

o We propose a GDCS model where [[U is a special case. 

o The theoretical bound on the required number of measurements of the GDCS model is 
obtained. 

o To solve the necessity of a priori-knowledge, which is a burden of the decoder, we propose 
a novel algorithm that iteratively detects the signals with the proposed algorithm. 
The remainder of this paper is organized as follows. We summarize the background of CS 
briefly in Section II. In Section III, we explain the concept of the existing joint sparse signal 
model and define its general version extension. Based on this model, we obtain the theoretical 
bound on the required number of measurements and propose a novel algorithm to capitalize on 
the GDCS in a practical environment in Section VI. In Section VII, numerical simulations are 
provided, followed by conclusions in Section VIII. 

II. Compressive sensing background 

When we deal with the signals sensed in the real world, in many cases we can represent a 
real value signal x G as sparse coefficients with a particular basis = [ipi, ...,ip]y]. We can 
write 

N 

x = ^V^„cu(n) (1) 

n=l 

where cu (n) is the nth component of sparse coefficients to and ipn is the nth column of the 
sparse basis. Without loss of generality, let assume that \\uj\\q = K. Here, ||a;||Q is the number 
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of nonzero elements in vector u. In matrix multiplication form, this is represented as 

X = *a; (2) 

Including the widely used Fourier and wavelet basis, various expansions, e.g., Gabor bases lfT2l 
and bases obtained by Principal Component Analysis (PC A) lfT3l . can be used as a sparse basis. 
For convenience, we use the identity matrix I for a sparse basis ^ . Without loss of generality, 
an arbitrary sparse basis is easily incorporated into the developed structure. 

Candes, Romberg and Tao [Q and Donoho lfT4l showed that a reduced set of linear projections 
can contain enough information to recover the sparse signal. This technique introduced in [|5l, 
[fT4| has been named CS. In the CS, a compression is processed by simply projecting a signal onto 
measurement matrix $ G M^^^^ where M <^ N . We can describe the compression procedure 
as follows. 

y = $x 

where y G M 

Since the number of equations (M) is smaller than the number of values (A^), this system is 
ill-posed. However, the sparsity of the signal allows perfect recovery if the restricted isometry 
property (RIP) of $ jSl, [[TSl is satisfied with an appropriate constant. Assuming that the signal 
X can be represented as in the sparsest coefficient vector Cj can be found by solving the 
following Iq minimization. 

Cj = argmin ||ci;||q s.t. y = (4) 

If the original coefficient vector to is sparse enough, there is no other sparse solution that satisfies 
y = ^^oj except for co = cu, which implies we can recover the original signal in spite of the 
ill-posedness of the system. 

However, although Iq minimization problem guarantees significant reduction in the required 
number of measurements for recovery, we cannot use Iq minimization practically because of 

f n\ 

its huge complexity. To solve the /q minimization, we must search possible sparse 

subspaces, which makes Iq minimization NP-hard [I16II . 

Instead of solving Iq minimization, we can use the solution of h minimization as the coefficient 
vector of the original signal, paying more measurements Q as a cost of a tractable algorithm. 

u = argmin ||a;||^ s.t. y = (5) 
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This approach is called Basis Pursuit. Contrary to Iq minimization, we can solve li minimization 
with bearable complexities, which is polynomial in A^. 

Not only norm minimization, but also an iterative greedy algorithm can be used for finding 
the original signal from the observed signal. The Orthogonal Matching Pursuit (OMP) [[TtI is 
the most typical algorithm among iterative greedy algorithms. It iteratively chooses the vector 
from the measurement matrix that occupies the largest portion in the observed signal y. It 
is proven in ifTTl that the original signal can be recovered with appropriately high probability by 
OMP. A greedy algorithm has been developed to more sophisticated algorithm, e.g., CoSaMP 
ifTSll and Subspace Pursuit GH. 



III. Joint Sparse Signal Model 

In |[T1, the joint sparse signal model is defined. Using the same notations with [[T]|, let A : = 
{1,2, J} denote the set of indices of signal ensembles. The ensembles consist of the signal 
j E A. We use (n) as the nth sample in the signal j. Each sensor j is given 



oN 



a distinct measurement matrix $j G 



i)MiXN 



, which is i.i.d. Gaussian matrix. The compressed 



signal Yj G M ^ can be written as yj = $jXj. Concatenating all the signals from 1 to J, we 



can write it in the following form. 



X 



Xl 




yi 


X2 


, Y = 










. yj . 



and $ 



$1 

$2 










(6) 



where X G M-^^, Y G M^^^ and $ G W^'' . Finally we can write 



Y = $X 



(7) 



In |[T1, Xj can be decomposed into two parts. The first part is common information zc, which 
is measured by every sensor, and the second part is innovation information zj, j G A, which is 
uniquely measurable by the sensor j. The signal x^ can be written accordingly as 



Zc + Zj, j G A 



(8) 



While ([8]) is composed of two kinds of components, we can refine the model by defining the 
partial common information as follows : It is the information measured by p multiple sensors 
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Fig. 1. Venn diagram description of the characterizing signal model for the DCS 

where p is an arbitrary number that satisfies 1 < p < J. Then, the innovation information in 
dH]) can be decomposed into partial common/innovation information. For ease of explanation, 
we consider a simple sensor network in Fig. [T] where each sensor measures its own innovation 
information and the full common information of all three sensors. In addition to those, three 
different partial common information can be measured in pairs as {sensor 1, sensor 2}, {sensor 
2, sensor 3}, and {sensor 3 and sensor 1}. For ji, j2 G A and ji ^ j2, let zc^-_^ denote partial 
common information measured by the sensors {ji, j^}- To avoid confusion with the notation 
of the existing signal model, we change the notation of innovation information to Zj^ , j G A in 
our model. With the defined notion, we can write the signal Xj of Fig. [T]in the following form. 

Xi = Zc + Zqi^2} + ZC{i,3} + 

X2 = Zc + ZC{2,i} + ZCf^,:,} + 2*2 (9) 
X3 = Zc + Zcj3,2j + ZC{3_,j + Zi3 

We can readily extend ^ to the case of an arbitrary large number of sensors. 

Adopting the same notation in DCS [[T]| which decouples the location and value of a signal, 
we also write an arbitrary sparse signal x as 

X = (10) 

for X G satisfying ||x||q = K, where 9 G M^^, called a value vector, contains only nonzero 
elements in x, and P G M^^^ , called a location matrix, is an identity submatrix, which consists 
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of K column vectors chosen from an x identity matrix. With these, we can describe the 
signal model ^ as follows. 



X 



1 T 



X = PO, where 



P e 

G 



E 



Ji,J2>ii7^i2 i=i 



(11) 
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where, for j,jij2 e {1, 2, 3} and ji ^ j2, Kc (P), ^C{,^,,v,} ^'^^ denote the sparsity 

of zc, ZC{^^ and Zj^ respectively. Likewise, Pc^ -Pc{j^ j^} ^'^^ -^*i ^'^^ location matrices and 
^C{j^ J2} ^*i ^'^^ value vectors of zc, '^C{j^ j^} ^'^^ ^*i respectively. From hence, we use 
P and 6 universally, not to be restricted to a specific signal model. 

IV. Theoretical Bound on the Required Number of Measurements 

In this section, we find the condition on the number of measurements to recover the original 
signal ensembles X in a noiseless environment. First, we summarize the theoretical bound of 
the existing DCS [1] and then obtain the theoretical bound for the simple three sensors network 
described in Fig. [T] in the view of the proposed GDCS model, which is followed by extension 
to the general case. 

If ||x||q = K, the minimum number of measurements to recover the signal x is 2K [[5l. If we 
know the supports of the elements, it is obvious that K measurements would be sufficient for 
perfect recovery. Therefore, thinking naively, the required number of measurements for recovery 

XI ll^ill assuming the known supports. However, in the DCS scenario, because full common 
information is measured by every sensor, it is possible to recover the original signal with the 
number of measurements less than ^ ll^illo- Then the remaining problem is how to allocate 
the measurements to sensors to prevent from missing the information. Obviously, when we have 
Zc {n) ^ and Zj [n) ^ simultaneously, we cannot recover both from a single measurement. 
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However, -lq (^) can be recovered with help of other sensors whose (n) does not overlap 
with the innovation information. In this notion, size of overlaps for a subset of signals F C A 
can be quantified. 

Definition 1 (|[T]|, Size of overlaps). The overlap size for the set of signals F C A, denoted 
as Kc (r^P)- the number of indices in which there is overlap between the common and the 
innovation information supports at all signals j G F'^ .• 

Kc (F, P) = |{n G {1, N} \zc (n) ^ and z, (n) ^ for Vj G F^}| (12) 

We also define Kc (A, P) = Kc (P) and Kc (0, P) = 0. 

Simply, Kc (F,P), for F C A implies a penalty term regarding the cardinality of indices of 
common information which should be recovered with help of measurements in F due to overlaps 
between common and innovation information at F'^. With the above definition, the theoretical 
required number of measurements for recovering the original signal can be determined from the 
following theorem. 

Theorem 1 ([[H, Achievable, known P). Assume that a signal ensemble X is obtained from a 
common/innovation information JSM (Joint Sparsity Model). Let M = (Mi, M2, Mj) be a 
measurement tuple, and be random matrices having Mj rows of i.i.d. Gaussian entries 

for each j G A. Suppose there exists a full rank location matrix P G Pf (^) where Pp {X) is 
the set of feasible location matrices such that 

J2 M, > i^eond (r, P)=IY1 (P) ) + (r, P) (13) 

jer Vjgr / 

for all F C A. Then with probability one over {^j}.^^, there exists a unique solution O to the 
system of equations Y = $P9; hence, the signal ensemble X can be uniquely recovered as 

X = pe. 

A. The three sensors network using the proposed GDCS model 

The theoretical bound in the proposed GDCS model can be computed in a similar way to 
Theorem [B We find the required number of measurements in a subset F C A. A difference 
between the existing DCS model and the proposed GDCS model is that there would be various 
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Fig. 2. Three possible cases of a subset F including sensors. 



types of overlaps among the signals in GDCS since we consider partial common information 
between them. 

Before we go into further detail of the bound, we define the notation of partial common 
information. We denote z^j^ as partial common information observed by a set of sensors 11 of 
cardinality 1 < |n| < J. For example, if partial common information is measured by a sensor set 
{ji, j2, Ja}, where A is an arbitrary number less than J, it can be represented as where 
n = {ji, ^2, Ja}- For the three sensors network considered in Fig. [2l all the partial common 
information can be written as zc^^^^^^ .^^ where ji, j2 G {1, 2, 3} and ji 7^ j^- Now we define two 
groups of information for explaining the theoretical bound. We divide all existing information 
into two groups, where the existing information includes full common, partial common and 
innovation information. 

Definition 2 (Exclusive information group). If the set of all sensors measuring given information 
is a subset of T, where F C A, such information is categorized into VLi (F). We can write this 
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as follows. 



fii (r) 



{z,^, |j n r = J, J e A} 
u{zcn|nnr =n} 
u{zc|Anr = A} 



(14) 



We call the defined group an exclusive information group since the information included 
in this group only can be measured from the sensors belonging to T. This concept can be 
clarified using Fig. [21 where each type information is symbolized, and three possible cases of 
a subset T are shown. In Fig. [2l-(a), full common information z^, partial common information 
zc{i 2}' ^C{2 zc^i and innovation information Zj^, Zj^, Zjg are all included in Vli (F). In Fig. 
|2]-(b), partial common information zc^-^^}' innovation information Zj^ and Zj^ are included 
in f2i (r). In Fig. l2l-(c), only innovation information Zj^ is included in fii (F). 

On the contrary to this, we can define another group as follows. 

Definition 3 (Shared information group). If the set of all sensors measuring given information 
has a nonempty intersection set with F, where F C A but is not a subset of F, such information 
is categorized into Q2 (r). We can write this as follows. 



We call this group a shared information group since the information included in this group 
can be measured from the sensors both belonging to F and not belonging to F. In Fig. |2]-(a), 
none of the information is included in (F). In Fig. l2l-(b), full common information zc and 



information zc, partial common information zc^^^y, and z^^^ are included in ^2 (r). Lastly, 
we define the third group as follows. 

Definition 4 (Unrelated information group). If the set of all sensors measuring given information 
has an empty intersection set with F, where F C A, such information is categorized into ^3 (F). 



fi2 (r) 



u {zcn |n n F 7^ 0, nnF^n} 

U{zc|AnF^0, AnF^A} 



(15) 



partial common information z^^j gp z^^^ 3J are included in ^2 (F). In Fig. [2l-(c), full common 



November 29. 2012 



DRAFT 



11 



We can write this as follows. 

{z,^ |jnr=0, jgA} 
fi3(r)= u{zc;jnnr=0} (16) 
u{zc|Anr = 0} 

Since ^3 (F) is not used in obtaining the theoretical bound, the third group has no practical 
meaning. Defined three groups are disjoint. 

For fix (F), since the information included in f2i (F) can be recovered only from the measure- 
ments of the sensors belonging to F, we must have the measurements of the sensors belonging 
to F as many as sparsity of the information included in f2i (F). On the other hand, for (r), if 
there is no overlap, we do not need to have the measurements of the sensors belonging to F since 
the information included in ^2 (r) can be recovered from the measurements of the sensors not 
belonging to F. However, if there is an overlap, the information cannot be recovered from the 
measurements of the sensors not belonging to F, so the measurements of the sensors belonging 
to F are needed. Therefore, we need additional measurements of the sensors belonging to F to 
compensate these overlaps. 

Now, we explain the concept of overlap in more detail using Fig. [21 Assume that we want to 
find the number of measurements required in a subset F C A for recovery. We assume that F = A 
as in Fig.[2l-(a). In this case, zc, zc^^ 2}' ^^{2 3}' ^^{i 3}' ^n' ^^'^ ^ia included in Hi (F), 

and (F) is empty. Therefore, as mentioned above, the required number of measurements is 
as follows. 

jer h,32,h¥'h j (17) 

whereF = A 

The right side of inequality in (flTl) is the sum of the sparsity of the information included in 
ill (r)- Next, let us consider the case of Fig.[2l-(b). In this case, z^^^ 2}' '^h^ '^12 included 
in r2i (F), and zc, z^^^s}' ^i^d ^^"{13} ^'^^ included in 1^2 (F). If there is no overlap on the 
information included in ^2 (r), we need the measurements of the sensors belonging to F for 
ZC{^2}' ^n' ^'^d since other information can be recovered from the measurements of the 
sensors not belonging to F. Therefore, the required measurements are as follows. 
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jer j=i (18) 

where F = {1,2} 

The right side of the inequality in (fTSi) is the sum of the sparsity of the information included 
in (r). However, assuming that there are overlaps on the information included in ^2 (r), 
e.g., Z(7^2 3} ('^) 7^ ^<^{i3} ("^^ 7^ some arbitrary n, more measurements than (fTSi) are 
needed since the overlapped information has to be recovered with the help of the measurements 
of the sensors belonging to T. The necessary number of measurements is as follows. 

E M, > Kc^^ (P) + E K,^ (P) + Overlaps 

ier ' j=i (19) 

where F = {1,2} 

where Overlaps denotes additionally required number of measurements. In the case of (c) in 
Fig. [2l similar to (fT9l) . the necessary number of measurements is as follows. 

Mj > Ki^ (P) + Overlaps 
jer (20) 

where F = {1} 

where Overlaps denotes additionally required number of measurements. In summary, in order 
to recover the original signal perfectly, we need measurements of the sensors belonging to F 
as many as the sparsity of the information included in Vti (F) plus the size of overlaps of the 
information included in VL2 (F). 

To calculate the theoretical bound on the required number of measurements analytically, we 
need to define the size of overlaps. Two types of overlaps can be differently considered: the 
overlaps of full common information and the overlaps of partial common information. We define 
the size of each type of overlap as follows. 

Definition 5 (Size of overlaps of full common information). Overlap size of full common 
information for the set of signals F C A, denoted as Oq (F, P), is the number of indices in 
which there are overlaps of the full common information and other information supports at all 
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signals j E V^. 

Oc(r,p) = ... 

{ne {l,...,iV}|zc(n) ^0 and z,^. (n) ^ 0, VjGr^} 

U |n G {1, N} zc (n) 7^ and zq^,^ (n) 7^ for ji, such that T'^ C {jija}} 
We also define Oc (A, P) = Kc (P) anJ Oc (0, P) = 0. 



(21) 



Next, we need to quantify the overlaps of partial common information. Using the same 
principle as in Definition [2l we can define size of overlaps of partial common information 
as follows. 

Definition 6 (Size of overlaps of partial common information). Assume that ji E T, and j2 E F'^. 
For the set of signals F C A, overlap size of partial common information measured by the signals 

{ill 32} i-^- 2C{3i ' ^^^'^^^^ ^^{3132} (■'^'■^)' '■^ number of indices for which there is a 
overlap between the partial common and the other information supports at a signal j = ja- 



(22) 



[nE{l,...,N}\zc^^^^^^{n)^0 and z,^^(n)7^o} 

U |n G {1, A^} zq^.^^^^j (n) ^ and zq^,^ (n) ^ for ^ ji, jg ^ ja} 
where ji G F, j2 ^ F 
We also define 0^^,^ ^^^ (A, P) = Kc,^^^^^^ (P) anJ Oc^^.^ ^^j (0,P) = 0. 

The reason that we only consider the overlap of partial common information which satisfy 
ji E F and ja ^ F'^ is to consider the size of overlaps for partial common information included 
in 1^2 (F). 

With these definitions, we can decide the theoretical bound on the number of measurements 
for the three sensors example in the proposed GDCS model. 

Theorem 2 (Achievable, known P). Assume that a three signal ensemble X is obtained from a 
full common/partial common/innovation information JSM (Joint Sparsity Model), as described in 
Fig. Ul Let M = (Ml, M2, M3) be a measurement tuple, and let {^j}^^^ be random matrices 
having Mj rows of i.i.d. Gaussian entries for each j E A. Suppose there exists a full rank 
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location matrix P G {X) where V p {X) is the set of feasible location matrices such that 



E M, > Oc (r, p) + E Oc,,,.,,> (r, p) + ... 
E i^c,,,,,, (P) + E (P) 



(23) 



for all r C A. Then, with probability one, there exists a unique solution to the system of 
equations Y = ^>P6 ; hence, the signal ensemble X can be uniquely recovered a* X = PO. 



Assuming a subset F C A, arbitrary partial common information zc 



in ,32} 



is included in 



Vti (r) if ji G r, j2 G r, and it is included in 0.2 (F) if ji G F, j2 G T'^. Therefore, for 
the partial common information 'Z'C^^^ ^hich satisfies ji G F, j2 G F, we consider the 
sparsity as the required number of measurements, and for the partial common information z^^^^ .^^ 
which satisfies ji G F, j2 G F'^, we consider the size of overlaps as the required number of 
measurements. 

B. The general case 

The case of a larger number of sensors can be readily extended from the three sensors networks. 
With a larger number of sensors, many various kinds of partial common information can be 
characterized depending on how they share the information. Now, we generalize the size of 
overlaps to derive the bound on the number of measurements. 

Definition 7 (Size of overlaps of full common information, the general version). The overlap 
size of full common information, denoted as Oc (F, P), is the number of indices in which there 
is overlap between the full common and other information supports at all signals j G F'^. 

Oc (r,p) = ... 

{ne{l,...,N}\zc{n)^0 and z,^, (n) ^ 0, VjGF'^} 

Zc (n) 7^ and '^Cuf. (^) 0; such that F'^ C [J j ^24) 
Zc (n) 7^ and zc^^^ (n) ^ 
andz,^ (n) ^0, Vj G F= - {\}Vl^, 

where is a sensor set for partial common information Zc^ and IjFIfc is a union of sensor 

k 

sets for partial common information. We also define Oc (A, P) = Kc (P) and Oc (0, P) = 0. 
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(l24l) consists of a union of three sets. The sets present overlaps between the full common 
information and the innovation information, overlaps between the full common information and 
the partial common information, and overlaps between the full common information and both of 
the innovation information and the partial common information, respectively. Now, Definition [6] 
should be extended to the general case. 

Definition 8 (Size of overlaps of partial common information, the general version). The overlap 
size of partial common information measured by a sensor set IT, i.e., 7.c^, denoted as Oc^i (r, P), 
is the number of indices for which there is overlap between the partial common and other 
information supports at all signals j G (H fl F^). 

Ocn(r,P) = ... 



{n e {1, N} |zcn (n) ^ and z,^. {n) ^ 0, Vj G (H n T^) } 




where 11^ is a sensor set for partial common information zq^ and [J Uk is a union of the sensor 

k 

sets for partial common information. We also define Oc^ (A, P) = Kcn (P) (0; P) = CI- 

As in (l24l) . (l25l) consists of union of three sets, and each set is a case of overlaps. The first set 
is overlaps between the partial common information and the innovation information, the second 
set is overlaps between the partial common information and other partial common information, 
and the third set is overlaps between the partial common information and both of the innovation 
information and other partial common information. 

With these definitions, we can compute the theoretical bound on the required number of 
measurements for the proposed GDCS model for the general case. 

Theorem 3 (Achievable, known P). Assume that a signal ensemble X is obtained from a 
full common/partial common/innovation information JSM (Joint Sparsity Model). Let M = 
{Ml, M2, Mj} be a measurement tuple, and let be random matrices having Mj 

rows of i.i.d. Gaussian entries for each j G A. Suppose there exists a full rank location matrix 
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P G Pi? (X) where P p {X) is the set of feasible location matrices such that 

EM,>Oc(r,p)+ E Ocn(r,P)+ ... 

E ^Cn (P) + E (P) 

zcneni(r) ier 

for all r C A. Then, with probability one, there exists a unique solution to the system of 
equations Y = $P0 ; hence, the signal ensemble X can be uniquely recovered as 'K = PO. 

The proof is described in the Appendix. As in Theorem [21 for the information included in 
fii (r), we consider the sparsity of the information as the required number of measurements. 
On the other hand, for the information included in ^2 (r), we consider the size of overlaps of 
the information as the required number of measurements. 

We can observe that Theorem [B the theoretical bound of the existing DCS model, is a special 
case of Theorem[3l i.e., the case not considering partial common information. Therefore, Theorem 
|3] can be regarded as a more refined version of Theorem [T] When P is unknown, it is known 
that additional |r| measurements in the right side of (l26l) would be sufficient for recovery 

V. Iterative Signal Detection With Sequential Correlation Search 

In this section, we discuss a method that can benefit from partial common information without 
any a priori-knowledge about correlation structure, which is the main obstacle of exploiting 
partial common information in practical implementation. To compare the requirement of a priori- 
knowledge of the existing DCS and the proposed GDCS, the problem formulation of the existing 
DCS model is described as follows. The notations of the information follow the existing DCS 
style. 



Z := 



X 




T T 
^1 ^2 


T 

■ 


GM^^ 






T T 
^1 ^2 










= Zc + Zj, 


where j G A 




$1 


$1 ■ ■ 


■ 








$2 ■■ 


■ 


^ ^JMx(J+l)N 




■ ■ 


■ 





(27) 

(28) 
(29) 

(30) 
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iWiz'ill, 



|W,z',||, 



(31) 
(32) 



Z = argmin ||Wcz'(^||^ 
5.1 Y = <I>Z' 

where W^; and W^, j G A are weight matrices. Thanks to a joint recovery, the improved 
recovery performance can be obtained compared to separate recovery. 

To get some insight on the proposed algorithm, let us consider a case in which partial common 
information is measured by a set of sensors A\{1,2,3}. This case can be formulated as the 
following problem by using the proposed GDCS model. 



X 



Z := 



«2 



e where H = A\ {1, 2, 3} 

z, if j ^ n 



$1 

$2 









$3 









$4 



$7 




Y = <I>Z 



Zi, if J e n 







$j 



t,JMx(J+l)N 



W- z' 



(33) 
(34) 

(35) 



(36) 



(37) 
(38) 



Z = argmin ||WcnZcnlli 
s.t. Y = ^Z' 

where yVcn and Wj^, j E A are weight matrices. As shown above, to exploit partial common 
information, we have to find a sensor set for partial common information 11, in this case 
A\{1,2,3}. Unfortunately, it is not straightforward to find which sensors are correlated. Since 
each sensor compresses its signal without cooperation of other sensors, there is nothing we can do 
to determine the correlation structure in a compression process. In a recovery process, although 
we can find the correlation structure by an exhaustive search, it demands approximately 2"^ 
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number of searches, which is not practical. Therefore, we need an moderately complex algorithm 
that finds the correlation structure. 

A novel algorithm is proposed for finding the correlation structure. The algorithm iteratively 
selects the least correlated sensor so that we can approximate the sensor set for partial common 
information 11. For simplicity, we assume a joint sparse signal ensemble X with partial common 
information zcj^, where 11 = A\{1,2,3} as in (|34|) . However, since we have no knowledge on 
the correlation structure, we cannot formulate the measurement matrix as in (|36l) . Instead, we 
use the solutions of the separate recovery and of the existing DCS framework which considers 
only full common information. The solution obtained by using the existing DCS has to include 
full common information, although the given signal ensemble X may not have full common 
information. By using this forcefully found full common information, we can obtain a clue about 
the correlation structure. When we compare li norm of the innovation information between the 
solution vector of the separate recovery and the existing DCS, while the li norm tends to increase 
(/i norm of the innovation information of the solution vector obtained using the existing DCS 
becomes larger.) if the corresponding sensor j ^ 11, it tends to be decreased if the corresponding 
sensor j G 11. 

Though this phenomenon is difficult to understand at first glance, it is quite straightforward. 
We should note that the forcefully found full common information may have some relation with 
the true partial common information. Actually, the forcefully found full common information 
is likely to be similar to the partial common information to minimize li norm of the solution 
vector. (But not always. We will explain it after this paragraph.) Then, if the sensor j E 11, which 
means it is one of the sensors that measure the partial common information, a joint recovery 
process successfully divides the energy of the signal into a joint recovery part (the first column 
of $ in (l30l) ) and a separate recovery part (the rest of the columns of $ in (l30l)). However, if 
the sensor j ^ 11, which means it is one of the sensors that do not have the partial common 
information, the innovation information of the sensor j ^ 11 must be made to compensate the 
forcefully found full common information, causing the increase in li norm of the innovation 
information. 

However, it can be exploited only if forcefully found full common information is similar to 
partial common information. If only a small number of sensors can measure partial common 
information, i.e., |n| is small, the forcefully found full common information has no relationship 
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to the partial common information. In this case, we cannot expect to find the sensor set 11 
based on the above observation. Therefore, in this paper, we assume that any partial common 
information can be measured by a sufficient number of sensors. This assumption can be justified 
by the fact that significant performance gain by joint recovery of partial common information 
can be achieved when a sufficient number of sensors measure the partial common information. 

Exploiting the above intuition, an iterative signal detection with a sequential correlation search 
algorithm is proposed with an underlying assumption on arbitrary inter-signal correlation (full 
common information or partial common information). The algorithm assumes the following. The 
number of sensors is J, each signal size is N and arbitrary inter-signal correlation (full common 
information or partial common information) exists. The received signal is denoted by Y G M^^"^, 
where M is the number of measurements assigned to each sensor node. 



Algorithm 1. 

0: Set the iteration counts ii and 12 to zero and initialize a matrix li^ - 

Define a matrix H (j) G M^^"^^^, j G A such that it holds $j on the jth block position while 
setting other blocks to zero. For example, 

1 T 



H(j) 







T 



0^ 



(39) 



Initialize a set of indexes of the sensors = 0, variables = and = and a matrix 



update 



n 

update 



= 0. The and f3i^ variables will be used as a parameter for escaping the loop and 
will be used as a temporary matrix for an updated measurement matrix. The notation 



"<^=" means to substitute the left hand side parameter with the right hand side parameter. 
1: Construct two measurement matrices as follows. 



«2 



$1 

$2 










$1 

$2 










(40) 



2: Obtain Zi and Z2 by solving the following weighted li minimization problems. 

Zi = argmin ||WiZi||-^ 
s.t. Y = ^iZ, 



(41) 
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Tj2 = argmin ||W2Z2||;^ 
si. Y = I>2Z2 



(42) 



where Wj, j = 1 or 2 are weight matrices. To avoid confusion, we denote the innovation 
information part of each solution in the following forms. The innovation information part 
indicates a part of the solution vector that is multiplied by the block diagonal part of 
or l>2. 



Zi 
Zs 



T 



t>Nxl 



pNxl 



j eA 
J e A 



(43) 



3: tti.+i 



Zi 



(We treat Zi (n) as zero if 



Zi in) 



4: Find the sensor index j* by solving the following problem. 

j* = argmin llz^ll — llzjll 
iijiii iijiii 

s.t. j e A- 

We denote the obtained index as jmin- 
5: 1^2+1 ^ - H 

$1 ••• 
$2 ••• 



(44) 



6-- '^Ui 



t2 + l 







7: Si2+i I jmin j. 



8: i2^i2 + 1. 

9; Go to step 2 and repeat the loop until the following two conditions are satisfied. 

I <i2 



(45) 



ctiy-i < ce. 



«2 



10: <= a.i^^i. 



11: 



update 



$1 T 



12: + 1, 22 ^ 0. 



$T ... $T 



74. With <|)"P'^°*'^^ update each matrix as follows. 
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$1 ^ I- 

L ' 1 J (46) 

75; Go to step 2 and repeat the loop until the following conditions are satisfied. 

I <ii 

(47) 

16: Estimate the signal by solving the following problem. 

Z = argmin ||WZ||-^ 



s.t. Y = <|^f^f Z 



(48) 



The algorithm consists of two phases, the inner and outer phases. In the inner phase, starting 
from an assumption of full common information, we exclude sensors one by one from the 
candidate sensor set, based on (l44l) . In the outer phase, we search out different types of inter- 
signal correlation. 

Intuitively, as the algorithm proceeds, we pursue the measurement matrix corresponding to a 
more sparse solution so that the algorithm provides better performance. However, computational 
complexity linearly increases with the number of iterations. In the worst case, the proposed 
algorithm needs J iterations in each inner phase. For the outer phase, since the number of 
columns of the measurements matrix is increased by one with every iteration, we can describe 
the complexity of our algorithm as O [J{N{J + yu))'^) where fi is the number of iterations of the 
outer phase in the algorithm. If the algorithm searches the correlation structure exactly, /i would 
be the number of inter-signal correlations. The existing DCS has complexity O ((A^(J+ 1))^). 
In the real simulation, however, the decoding time is necessarily limited since the stopping 
criterion is the approximated Iq norm of the solution vector which is a natural number. The 
iteration continues only if the approximated Iq norm of the solution vector is reduced compared 
to the previous iteration, which implies that the maximum iteration number is the approximated /q 
norm of the solution vector of the first iteration. The real CPU times consumed by the algorithm 
are compared in the next section. 

VI. Numerical Experiments 

In this section, we demonstrate the GDCS through numerical experiments. Assuming various 
inter-signal correlations, we compare the detection performance when using Oracle-GDCS, 
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The Number of Measurement per Signal, M 



(a) 




The Number of Measurement per Signal, M 



(b) 

Fig. 3. Performance comparison of Oracle GDCS, GDCS with the proposed algorithm, the existing DCS and separate recovery 
when (a) K,^ = 4, Kcn = 6, |n| = 6, (b) K,^ = 4, A'cn = 4, |n| = 6 
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The Number of Measurement per Signal, M 



(a) 




The Number of Measurement per Signal, M 



(b) 

Fig. 4. Performance comparison of Oracle GDCS, GDCS witli tlie proposed algorithm, tlie existing DCS and separate recovery 
when (a) K,^ = 4, Kc = 6, Kc^^ = 6, Kc^^ = 6, |ni| = 7, [Hal = 6, (b) K,^ = A, Kc = 5, A'cn, = 3, Kc^^ = 
3, |ni| = 7, inal = 6 
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which means GDCS with a priori-knowledge of correlation structure, GDCS with the proposed 
sequential correlation search algorithm, the existing DCS and separate recovery. 

The simulation environment is as follows. Each signal element is generated by an i.i.d. standard 
Gaussian distribution, and the supports are chosen randomly. The signal size N and the number 
of sensors J are fixed to 50 and 9, respectively. As aforementioned, the identity matrix is used 
as a sparse basis without loss of generality. The measurement matrix is composed of i.i.d. 
Gaussian entries with a variance 1/M. We assume a noiseless condition in all simulations. The 
type of inter- signal correlation, the correlation structure (the number of sensors that measure 
partial common information) and the sparsity of the information are determined as simulation 
parameters, and the corresponding sensors involved in the correlation are chosen randomly. 

We use MATLAB as a simulation tool, and YALLl solver is used for solving the weighted li 
minimization. We use an iterative weighted h minimization method introduced in [|20ll to obtain 
adequate weight matrices within a reasonable time. The probability of estimation error within 



X-X 



the resolution is used as a performance measure where error is calculated by 
and the resolution is set to 0.1. 

In Fig. [3l-(a) and (b), GDCS with the proposed sequential correlation search outperforms the 
existing DCS when there exists single partial common information. Using the GDCS with the 
proposed sequential correlation search, the performance is almost the same as in the oracle case. 
According to the simulation, about 7 measurements are saved in (a) and about 5 measurements 
are saved in (b) comparing to GDCS with the proposed sequential correlation search and the 
existing DCS [IJ. This implies that we can obtain approximately 23% in (a) and 18% in (b) gains 
in the number of measurements when we use GDCS with the proposed sequential correlation 
search. The gap between (a) and (b) is caused by different partial common information sparsity 
environments. 

We calculate the consumed CPU time when the individual number of measurements are 25, 
30, and 35 each. The CPU time is counted only in the DCS and the GDCS with the proposed 
algorithm. We average the CPU time over 100 different realizations with the simulation setting 
associated with Fig.[3]-(a) and (b) each. The units of CPU time are seconds. In the (a) environment, 
the DCS CPU times are 1.42, 1.33, 1.32, respectively, while the GDCS with the proposed 
algorithm CPU times are 4.07, 3.92, 3.89, respectively. In the (b) environment, the DCS CPU 
times are 1.10, 1.08, 1.12, respectively, while the GDCS with the proposed algorithm CPU 
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times are 3.52, 3.30, 3.36, respectively. Differently from the big-0 comparison which considers 
the worst case, we can observe that significant performance improvement can be achieved with 
reasonable time. 

In Fig. |4l-(a) and (b), which consider multiple inter-signal correlation including full common 
information and two kinds of partial common information, GDCS with the proposed sequential 
correlation search still provides a better performance than the existing DCS [[U. However, unlike 
Fig.|3l there is a performance gap between Oracle-GDCS and GDCS with the proposed sequential 
correlation search. We conjecture that, since partial common information behaves as interference 
to other partial common information during a sequential correlation search, the algorithm fails 
to identify the correct correlation structure even though there is no noise. In fact, several missed 
detections and incorrect detections of correlation are observed in the proposed algorithm in some 
realizations. About 10 measurements are saved in (a), and about 4 measurements are saved in (b) 
when we compare GDCS with with the proposed sequential correlation search and the existing 
DCS UJ. In total, 26% and 13% gains in the number of measurements can be obtained in (a) 
and (b), respectively. We also calculate the consumed CPU time when the individual number of 
measurements are 25, 30, 35, 40, and 45. We average the CPU time over 100 different realizations 
with the simulation setting associated with Fig. |4]-(a) and (b) each. In the (a) environment, the 
DCS CPU times are 2.15, 1.98, 1.86, 1.86, 1.87, respectively, while the GDCS with the proposed 
algorithm CPU times are 34.17, 34.06, 28.92, 31.97, 29.79, respectively. In the (b) environment, 
the DCS CPU times are 1.42, 1.35, 1.40, 1.37, 1.36, respectively, while the GDCS with the 
proposed algorithm CPU times are 16.41, 16.43, 17.39, 16.56, 15.69, respectively. It is observed 
that the CPU time of the GDCS with the proposed algorithm increases significantly as the 
correlation structure becomes more complex. 

VII. Conclusions 

In this paper, we extended the framework introduced in [[U to a realistic environment. In the 
existing DCS model, partial common information must be considered as innovation information, 
which cannot be used in joint recovery. The proposed GDCS model refines the existing model 
so that it can use partial common information in the joint recovery process. In this notion, 
we proposed a framework of GDCS and obtained the theoretical bound on the number of 
measurements. We also proposed a detection algorithm to identify the correlation structure so 



November 29. 2012 



DRAFT 



26 



that it can exploit this information in joint signal recovery without a priori-knowledge. Numerical 
simulation verifies that the proposed algorithm can reduce the required number of measurements 
compared to the DCS algorithm. 

Future research should address a method of achieving performance close to that of Oracle- 
GDCS in the presence of a large number of different partial common information. 

Appendix 

Proof of Theorem [3] 

In the appendix, we prove Theorem [3l the theoretical bound of the number of measurements, 
the general version. Even though this proof follows the Proof of Theorem [3] in [[U, since the 
considered system setup involves partial common information, the proof is nontrivial and more 
complex. 

At first, we assume that there are three kinds of information, which are full common informa- 
tion, partial common information and innovation information. Furthermore, arbitrarily, A kinds 
of partial common information exist. Then we can write the matrix <I>P as follows. 











^lP^, 

















A 

s 





'^2P^, 














■ ■ • ^vecjOn^ Pcn^ 















_ ^jPc 















■ '^JP^J 



(49) 

where $^ec_Cn ' J = 1,...,A is a partially zeroed matrix corresponding to partial common 
information zc^j. For example, if A = 2, Cjii = A\ {1} and Cnj = A\ {1, 2}, 





^iPc 


^iPcu, 





^iPr, 
























^2P., 








$p = 


^sPc 


^sPcu, 


^■sPcu, 















_ ^jPc 


^jPcn, 


^jPcn, 











■ ^JP^J 



(50) 
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Using much of |[T1, we exploit a meaningful result from graph theory. Before proceeding 
further, we define an expression for simplicity. We write Ilinciude{j) to signify the partial common 
information set that includes the sensor index j. In other words, Ilinciude{j) represents every set 
among Hi, 11^ that includes j. Next, a graph is considered as in Fig. \5\ That graph has some 
assumptions and properties as follows. 

A 

o D is defined to be -D = Kc (P) + Yl ^^Cn (P) + X] which implies the joint sparsity 

j=i ' jeA 

of the signal model. 

o For every d E {1,2, Kc (P)} ^ Vv, such that column d of Pc does not also appear as a 
column of P, or of Pcr, , we have an edge connecting d to each vertex (j, m) G Vm 

include{j) 

for 1 < m < Mj. 

o For every d G [Kc (P) + 1, Kc (P) + Kc^^ (P)} C Vy, such that column d of Pg^i does 
not also appear as a column of of P, or of Pc„ , we have an edge connecting d to 

include{j) 1 

each vertex (j, m) G Va/ for 1 < m < M,-. 

o For every d G ( Ac (P) + E i^Cn, (P) + 1, (P) + E Acn, (P) j C 1^, 1< A; < A, 

I 1=1 /=i ' J 

such that column d of P^j^ does not also appear as a column of Pj or of P^j^ , we 

have an edge connecting c? to each vertex (j, m) G Va/ for 1 < m < Mj. 

o For every d G l-Ac- (P) + E A'cn, (P) + I5 •••5 d is measurable at the sensor j as 

innovation information, and we have an edge connecting d to each vertex (j, m) G Va/ for 

1 < m < Mj. 

For example, assuming that Kc (P) > 3, we can observe in Fig. [5] that edge 3 has no 
connection to (1, m), 1 < m < Mj. This is because there is an overlap between full common 
information and other information at the sensor 1. 

Now we are ready to exploit Hall's marriage theorem for bipartite graphs ll2n . Hall's marriage 
theorem states that, assuming an arbitrarily bipartite graph {Vi,V2,E), if the cardinality of an 
arbitrary subset of Vi is always greater than the cardinality of its neighbors in V2, every element 
of Vi has a unique neighbor in V2 with no overlap. 

In the graph G = (VV, Vm, E), let us consider an arbitrary subset tt, and let E (tt) denote the 
set of neighbors in Vm- To satisfy Hall's marriage theorem, \E {ti)\ > \n\ for any n. If we let 
S-K = { j G A m) E E (tt) for some m}, the following statement is required for assigning a 
unique neighbor to every edge in Vy. 
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Fig. 5. Bipartite graph for generalized distributed compressive sensing. The graph G = {Vv, Vm, E) impHes the relationship 
between the value vector coefficients Q and the measurements Y. 



^M, >|7r| (51) 
Therefore, it would be sufficient to prove the following inequality. 



Oc(5^,P)+ Ocn(5^,P)+ ^Cn (P) + 5Z (P) > kl (52) 

Then, since the condition of Theorem [3] includes all subsets F G A, Hall's marriage theorem 
is satisfied for the graph G. To prove (|50l ). we divide tx into three disjoint parts, i.e., tt = 

TT/ U TXpc U Tic- 

First, as shown in [T], Itt/I < ^ Ki. (P) since all innovation information measured by S-,, 
are counted. Next, we claim that |7rc;| < Oc {S,,, P). This is because ttc C tt, and it implies that 
Oc (St,^, P) < Oc {St,, P). Consequently, |7rc;| < Oc {S.„, P). When we consider npc, we write 
Tipc = TTpc, UvTpc, u^is^), whcrc TTpc, Ujis^) contains partial common information included 
in TTpc and ^Ij (Sj,) simultaneously. Here, j = I, 2. In terms of vrpc Qi(5^), by the same reason 
of the case of innovation information, Ivrpc, | < -^Cn (P)- When vrpc, ^2(5^) 

is considered, by the same reason of the case of full common information, Ivrpc ^72(5^)! ^ 
Ocyi ('S'ttiP). Considering all of this, we claim that, if Theorem [3] is satisfied, every 
element in Vy has an unique matching in Vm- 
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Before proceeding to the next step, we introduce an important lemma. 

Lemma 1 Full rank property of a Gaussian or zero vector matrix). Let T^"^"^) be a {d — 1) x 
{d — 1) matrix having full rank. Construct d x d matrix T^'^^ as follow s: 



(53) 



where Wi, f 2 G R'^^^ are vectors with each entry being either zero or a Gaussian random variable, 
tu is a Gaussian random variable, and all random variables are i.i.d. and independent ofT^'^~^\ 
Then with probability one, T^^^ has full rank. 

The next step is equivalent with the method used in |[T|. Here, we briefly describe the process. 
At first, by subtracting the columns of innovation information from the columns of full common 
information and partial common information, all the same valued columns are removed, and a 
partially zeroed matrix is formed. Second, based on the obtained unique neighbor, the rows are 
chosen. For example, if the sensor j contains K neighbors, K rows are chosen within the part of 
the sensor j, i.e., the rows of the matrix corresponding to the sensor j. Finally, in the constructed 
matrix that consists of chosen rows in the second step, the columns are rearranged. In the order 
of the sensor index, the information component of which its neighbor is in the corresponding 
sensor is placed. For example, if the sensor j contains the neighbor of the A;th component of 
full common information, the A;th column of full common information part is stacked in the 
space of the sensor j. All these operations are column subtraction and column or row choice, 
which means there is no rank increase. Then, by the construction, all the diagonal elements of 
the constructed matrix are i.i.d. Gaussian random variables, and the other part is a zero or i.i.d. 
Gaussian random variable. Therefore, by Lemma [H <I>P has full rank. 
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